Skip to content

[rl] Add optional Verifiers rollout integration - #4356

Open
wwwjn wants to merge 11 commits into
mainfrom
verifiers-dapo-math
Open

[rl] Add optional Verifiers rollout integration#4356
wwwjn wants to merge 11 commits into
mainfrom
verifiers-dapo-math

Conversation

@wwwjn

@wwwjn wwwjn commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Add a verifiers example that reuses the DAPO Math dataset and training setup while delegating episode execution and reward calculation to Verifiers.
  • Keep all Verifiers-specific adapters inside the example so other TorchTitan recipes do not import or require verifiers.
  • Use the Verifiers null harness with a local subprocess runtime. No Docker or remote sandbox is required, and no tools are exposed to the model.
  • Leave the original dapo_math implementation unchanged as the native comparison path.

The example can be launched with:

python -m torchtitan.experiments.rl.train \
  --module verifiers \
  --config rl_dapo_qwen3_4b_verifiers_8k

100-step comparison

Both runs loaded Qwen3-4B-Base from the same HF checkpoint on four H100 GPUs and used identical overrides:

  • 100 optimizer steps
  • 2 generator GPUs and a TP=2 trainer
  • 4 prompts per step and 8 samples per prompt (32 training samples per step)
  • maximum completion length 2048 and maximum total length 10240
  • validation disabled
  • zero-standard-deviation reward groups retained
  • checkpoint load-only mode
  • W&B online logging enabled

The table reports the mean across all 100 W&B training steps unless otherwise noted.

Metric Native dapo_math verifiers Verifiers delta
Mean training reward 0.1878 0.1978 +0.0100
Mean truncation rate 7.53% 6.47% -1.06 pp
Mean response length 841.7 tokens 834.2 tokens -0.9%
Mean loss -0.00141 -0.00111 +0.00030
Mean step time 7.118 s 7.101 s -0.2%
Full-step throughput 4464.0 tokens/s 4562.7 tokens/s +2.2%
Forward/backward throughput 8722.0 tokens/s 8473.0 tokens/s -2.9%
Mean decode latency 13445.3 ms 13878.5 ms +3.2%
W&B total runtime 922.5 s 986.6 s +6.9%

The 2 DAPO math run shows similar reward and rollout:
Screenshot 2026-09-01 at 8 59 06 PM

Screenshot 2026-09-01 at 8 56 28 PM

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 27, 2026
@wwwjn
wwwjn requested a review from felipemello1 August 28, 2026 18:20
wwwjn
wwwjn commented Aug 28, 2026
View reviewed changes
Comment thread torchtitan/experiments/rl/examples/verifiers/rollouter.py

@dataclass(kw_only=True, slots=True)
class Config(VerifiersRollouter.Config):
train_dataset: VerifiersTaskDataset.Config = field(

@wwwjn wwwjn Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dataset is defined through verifiers

@wwwjn
wwwjn force-pushed the verifiers-dapo-math branch from 762930e to 36c417d Compare August 31, 2026 23:51
wwwjn
wwwjn commented Sep 1, 2026
View reviewed changes
Comment thread pyproject.toml
include = ["torchtitan*", "torchtitan_recipes*"]

[tool.setuptools.package-data]
"torchtitan.experiments.rl.examples.verifiers" = ["verifiers_env.toml"]

@wwwjn wwwjn Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because when release, python packaging includes .py files automatically, but not .toml files. This will not make verifiers a hard dependency

@wwwjn
wwwjn marked this pull request as ready for review September 1, 2026 16:18
@wwwjn
wwwjn requested a review from pzhan9 September 1, 2026 16:18
@felipemello1

felipemello1 commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

not a review, just a general comment: Its unclear to me what the examples/verifiers/... contains. Examples should be something like DAPO_math, search_r1, etc.

But it seems that you also need a components. Is this general for every verifier-based example? If so, maybe lets put it /src?

If this example is about verifiers_dapo_math, lets make it explicit in the folder. Later on we might have verifiers_swebench, for example. Another option could be to have verifiers/dapo_math and verifiers/swe_bench.

Also, it might be worth checking how other libraries integrate with verifiers. Example: https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/verifiers_rl

@wwwjn

wwwjn commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

not a review, just a general comment: Its unclear to me what the examples/verifiers/... contains. Examples should be something like DAPO_math, search_r1, etc.

But it seems that you also need a components. Is this general for every verifier-based example? If so, maybe lets put it /src?

examples/verifiers/componets is general for every verifier-based example. My original goal is to make this folder a stand-alone example of "titanRL + Verifiers". We only have one Verifiers examples now but I would want this design is easy for user who have their own sandbox account (eg, prime sandbox) to extend this example for agentic training, by reusing everything under examples/verifiers/componets.

If this example is about verifiers_dapo_math, lets make it explicit in the folder. Later on we might have verifiers_swebench, for example. Another option could be to have verifiers/dapo_math and verifiers/swe_bench.

verifiers/swe_bench sounds better. What about examples/verifiers/swe_bench, examples/verifiers/components, like what tinker did -- Put verifiers as a self-contained folder / recipe

Also, it might be worth checking how other libraries integrate with verifiers. Example: https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/verifiers_rl

In terms of folder organization, we are very similar. In-terms of integration point, I will take more look into details!

@felipemello1 felipemello1 mentioned this pull request Sep 1, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@felipemello1 felipemello1 Awaiting requested review from felipemello1
@tianyu-l tianyu-l Awaiting requested review from tianyu-l tianyu-l is a code owner
@fegin fegin Awaiting requested review from fegin fegin is a code owner
@wconstab wconstab Awaiting requested review from wconstab wconstab is a code owner
@pzhan9 pzhan9 Awaiting requested review from pzhan9

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ciflow/rl CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wwwjn @felipemello1